Repository navigation
GLM: fix KDA and router thread limits on M1 - #708
Conversation
434a62f to
588b02c
Compare
Decode launches hc_expand_10 at 1,024 threads a group at every step's last block boundary, and compiled offline for M1 and M2 the kernel built for 928. Its spec in tools/zig/gen_glm_kernels.py gains threads=1024, as #708 gave KDA and the router; regenerated from the Python kernels it came from, only hc_expand_10.metal changes, by its [[max_total_threads_per_threadgroup(1024)]] line, and offline it now builds for 1,024 on M1 and M2. The arithmetic is unchanged. launch_check.zig runs it at 1, 5 and 16 rows against the exact values (dyadic inputs keep every fp32 step exact, so the bf16 store is the one rounding) and checks the rows past the window and the unused outputs stay untouched. The kernel contract records its launch, with the M1/M2 limit its compiles show.
|
Thank you, @mrv777. Asking the compiler for the 1,024 threads these kernels launch with is the fix, and the capacity assertions with the 512-thread control make the fault testable on any chip. Your commit ships in TensorFold 1.0.12 under your name ( On an M3 Ultra all three launch_check tests pass. That chip's compiler already built these pipelines for 1,024 threads, so it cannot show the fault your M1 Max did; offline compiles for M1 and M2 can. Before your change both KDA pipelines compiled for 512 threads there and 13 of 16 router widths fell short; after it, every one compiles for 1,024. GLM-5.3-Flash with 8 of its 45 layers gave the same tokens as before, drafted and with The same offline check showed Posted by Ash's AI Assistant Synthena |
2 participants
On M1 Max, the two KDA pipelines compile for 512 threads, while several router specializations also fall below their 1,024-thread dispatch. These launches can leave outputs unwritten without a command-buffer error.
Reserve 1,024 threads in the generator and regenerate both KDA kernels and the router. Arithmetic, reduction order and dispatch sizes stay unchanged. Regressions check all 16 router row counts, both KDA head counts, output/state/cache coverage and recurrent window boundaries.
Rebased onto 1.0.10 at
f930fbb7. #704 is merged; this branch now contains only the KDA/router commit.Validated on M1 Max with Zig 0.17.0 (
-Dcpu=apple_m1):API Validation passes; both KDA tests also pass Shader Validation. The router’s unchanged instrumented memory exceeds M1’s 32 KiB limit in some variants. Full-model inference, two-Mac TP2 and other GPUs remain untested. Raw logs remain local.